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Abstract: The Circles of Care initiative emphasized the 
importance of developing an outcomes measurement plan 
that was consonant with the model system of care as well 
as community values and priorities. This analysis suggests 
that the Circles of Care grantees achieved this key 
programmatic objective, but that a major constraint was the 
tendency of funders, including the Substance Abuse and 
Mental Health Services Administration (the funder of Circles 
of Care), to mandate their own outcomes measurement 
plans. Funders are encouraged to balance their needs for 
commonality of measures across programs for their own 
evaluation purposes with the needs of service providers to 
utilize measures that meet their unique programmatic and 
community contexts. 


Mental health services are expected to demonstrate positive outcomes 
for the children, adolescents, families, and communities they serve (Nixon & 
Northrop, 1997). Indeed, the importance of demonstrating such positive 
outcomes for programs serving American I ndian and Alaska Native (AI/AN) 
communities was emphasized by many of the parent and community 
participants in the Circles of Care (CoC) planning process, who advocated 
for services that improved the mental health of their children and adolescents. 

However, parents, community members, program staff, and 
evaluators raised the following key questions throughout the CoC planning 
process: (a) what constitutes a positive outcome for AI/AN children, 
adolescents, and their families; and (b) how would these outcomes be 
measured? Underlying these questions was the concern that mainstream 
approaches to measuring outcomes for mental health services were 
inappropriate for programs serving AI/AN communities. The major weakness 
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of these mainstream approaches was their emphasis on clinical 
conceptualizations of mental health, with measurement focusing exclusively 
on problems rather than strengths. 

Fortunately, these concerns were anticipated in the CoC Guidance 
for Applicants (GFA), which identified explicitly the development of a plan for 
measuring outcomes as a key goal of the initiative (SAMHSA, 1998a). I ndeed, 
the introductory section of the GFA stated that "The program is intended to 
support tribes and urban Indian organizations in their efforts to develop 
service delivery models, which will generate the outcomes selected by 
American Indians/Alaska Natives for their own children [emphasis added]." 
Other sections of the GFA amplify this intention: 

The program is also intended to support the development 
of measures and processes that will be useful to tribal and 
urban I ndian organizations in evaluating their service models 
against the outcomes they have selected. 

Thus, the underlying message of the GFA was that strategic plans 
must include a plan for culturally and programmatically relevant approaches 
to measuring outcomes. Such a plan would assure that the model, once 
implemented, would be evaluated using the methods and measures consistent 
with its design, objectives, and values (SAMFISA, 1998a). 

I n this paper, we describe the framework, process, and products of 
this key component of the CoC evaluation. First, we describe the framework 
provided by the Circles of Care Evaluation Technical Assistance Center 
(CoCETAC) to the CoC grantees for developing their plan for measuring 
outcomes. Next, we describe the process the grantees used for developing 
their plans and a series of pragmatic issues that shaped this process. Then, 
using the framework provided by CoCETAC, we provide an overview of the 
grantees' plans for measuring outcomes. Finally, we analyze the process 
and products of this evaluation component and their implications for 
communities, evaluators, and policymakers. 

The Circles of Care Framework for Developing a Plan for Measuring 

Outcomes 

Grantees were presented with an idealized approach to develop a 
plan for measuring outcomes that reflected the values, objectives, and 
programmatic design of the strategic plan itself. CoCETAC identified the 
following five aspects of measurement for the grantees to consider as they 
developed their Outcome Measurement Plans: (a) "Domains of Measurement," 
(b) "Levels of Assessment," (c) "Assessment Approaches," (d) "Informants," 
and (e) "Timeline." To ensure that the outcome plans were not simply 
reflective of those measures that were most popular or expedient, grantees 
were encouraged to consider each of these aspects before choosing the 
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specific outcome measures they would employ. Each of these aspects of the 
Outcomes Measurement Plan is described in more detail below. 


Domains of Measurement 

First, the grantees identified areas or "domains" that would be 
impacted by their planned services. The following potential domains were 
identified by grantees and CoCETAC through the Needs Assessment activities 
described by Novins, LeMaster, J umperThurman, & Plested (2004) in another 
paper in this volume: local concepts of health and mental health, symptoms, 
indicators of health and dysfunction, resiliency and risk, tribal identities, 
spirituality, family profiles, availability of services, barriers to accessing services, 
and acceptability of services. Additionally, grantees were encouraged to 
develop other domains that were appropriate to their service area needs. 


Levels of Assessment and Assessment Approaches 

Next, the grantees determined the "levels of assessment" they would 
measure. Would they measure individual-based outcomes, family-based 
outcomes, and/or community-based outcomes? I n addition, grantees were 
asked to choose specific approaches to their assessments. Would they 
concentrate on measuring outcomes from a problem-based perspective, 
typical of the measures used for programs serving non-Indian/American 
I ndian and Alaska Native communities? Alternatively, would they also measure 
outcomes from a strength-based perspective, which was more consistent 
with AI/AN concepts of health and balance? Would they use some combination 
of these two approaches? The combination of these two aspects of the 
framework can be conceptualized as a 2 x 3 matrix as presented in Table 1. 


Table 1 

A Matrix of Levels of Assessment and Assessment Approaches 




Assessment Level 


Assessment Approach 

Individual 

Family 

Community 

Problem-Based 

IZZI 

IZZI 

IZZI 

Strength-Based 

IZZI 

IZZI 

IZZI 
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I nformants 

Next the grantees identified the informants they would utilize in 
measuring their outcomes. CoCETAC and the grantees generated a working 
list of potential informants. Identified informants were the following: the 
child/adolescent themselves, their parents/ caregivers, the extended family, 
elders, traditional healers, community members, project staff members, 
biomedical clinicians, and secondary data (e.g. county mental health, schools 
and juvenile probation). Project staff members were included in this list 
because of the multiple perspectives they bring, including that of parents, 
extended family members, and community members in addition to those of 
clinicians and planners. 


Timeline 

The grantees then produced a project timeline. Grantees were 
asked to consider when they would expect their programs to demonstrate a 
measurable difference in the domains they had identified and how long they 
would expect these impacts to last. For example, a baseline could be 
established as 'entry into the system' for measuring many aspects of problems 
and strengths at an individual or family level. Specific follow-up intervals 
could then be specified that matched the grantee expectations for meaningful 
changes. However, some outcomes might be difficult to match to an individual 
child or adolescent's entry into the System of Care. For example, information 
on school-wide test scores, suspensions, and rates of graduation would only 
be available on an annual basis consistent with school district or state 
reporting requirements. 


Selecting Specific Measures 

After specifying the aspects of the Outcomes Measurement Plan 
noted above, the final step for developing this plan was to select specific 
measures to employ. CoCETAC and the grantees developed a substantial 
library of potential measures which was revised several times as new 
measures were identified. 

The rationale for this approach of identifying the specific aspects of 
their plans prior to selecting specific measures was that grantees would be 
able to evaluate and choose potential measures based on utility for their 
community contexts and specific service delivery models rather than popularity 
in non-AI/AN programs and usage in county, state, and federal funding efforts. 
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Factors Affecting the Development of the Plans 

While the approach outlined above was considered ideal, three 
important logistical and political issues resulted in a more pragmatic approach 
for many of the grantees. First was the issue of time. The CoC evaluation 
effort is demanding and time-consuming. By the time the grantees reached 
this component of the evaluation (outcomes measurement), which was 
usually well into the third year of their grants, they needed to move 
expeditiously in order to meet their planning deadlines. Indeed, First Nations 
Community HealthSource wrote that the specific challenge met in completing 
this component of the evaluation was the "lack of time." 

Second was the issue of potential funding sources for implementing 
their plan. Many funders, including CMHS, require specific outcomes 
measures as a part of their agreement to fund services. For example, CMHS' 
own Comprehensive Community Mental Health Services for Children and 
their Families Program, which was identified by many of the grantees as an 
important potential source of funding, mandates an extensive Outcomes 
Measurement Plan (SAMHSA, 1998b). Many grantees were reluctant to 
develop a plan that they would be unable to use under this and other initiatives. 
Third, many of the service organizations that would participate in the grantees’ 
model systems of care had their own Outcomes Measurement Plans that 
would have to be incorporated into the grantees' plans as well. Many of 
these measures were mandated by federal, state, and third party funders of 
these programs. 

Therefore, many of the grantees decided to simultaneously review 
these existing measures and slot them into the aspects of their Outcomes 
Measurement Plan identified above. This enabled them to identify those 
outcomes that would not be measured by mandated instruments, and consider 
whether additional measures were needed to 'cover' these outcomes. Again, 
a pragmatic issue the grantees faced here was participant burden. With 
extensive measurement plans already in place or mandated by potential 
funders, the grantees had to decide whether additional measures would 
create undo burden for participants in their systems of care. 

We now review the characteristics of these plans. 

Characteristics of the Outcomes Measurement Pians 


Domains of Measurement 

Table 2 displays the domains of measurement covered by the 
grantees' Outcomes Measurement Plans. ^ Indicators of Health and 
Dysfunction and Resiliency and Risk were covered by all eight of the grantees 
that submitted plans for analysis in this paper. The domain of Symptoms 
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was covered by seven grantees. Three grantees added the domain of 
Satisfaction with Services to the original list. 


Table 2 

Domains of Measurement Covered by the Circles of Care Outcomes 
Measurement Plans 


Domains 


Number of Grantees I ncluding this 
Domain in their Plan 


Indicators of Health and Dysfunction 8 

Resiliency and Risk 8 

Symptoms 7 

Service System Needs (Availability) 6 

Spirituality 5 

Barriers to Accessing Services 5 

Acceptability of Existing Services 5 

Local Concepts of Health and Mental Health 4 

Tribal Identities 4 

Family Profiles 4 

Satisfaction with Services 3 

Acculturation (or Cultural Identity) 2 


Notes: Only those domains identified by two or more grantees are included in this Table. 


Levels of Assessment and Assessment Approaches 

These two aspects of the grantees plans are summarized in Table 
3. Grantees developed plans that were very balanced, both in terms of using 
problem- and strength-based approaches to measurement and in terms of 
assessing outcomes at individual, family, and community levels. I ndeed, the 
emphasis on measuring community-level outcomes is unusual for mental 
health service systems, but consistent with the broad goals of the grantees' 
plans and the CoC initiative as a whole. 


Table 3 

Levels of Assessment and Assessment Approaches I ncluded in the Circles 
of Care Outcomes Measurement Plans 




Assessment Level 


Assessment Approach 

Individual 

Family 

Community 

Problem-Based 

6 

6 

5 

Strength-Based 

6 

6 

6 
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I nformants 

The list of informants included by the grantees in their plans is 
summarized in Table 4. All grantees included parents/caregivers as 
informants in their plans; seven included children/adolescents themselves; 
five included extended family members and elders. There was considerable 
divergence in informants beyond these three, fewer than half the grantees 
including informants such as biomedical clinicians, secondary data, and 
traditional healers. 


Table 4 

Key informants I ncluded in the Circles of Care Outcomes Measurement 

Plans 


Domains 

Number of Grantees 1 ncluding this 

Domain in their Plan 

Parent/Caregiver 

8 

Child/Adolescent 

7 

Project Staff Member 

6 

Extended Family 

5 

Elder 

5 

Biomedical Clinician 

3 

Secondary Data 

3 

Community Member 

3 

Traditional Healer 

2 

Stakeholders 

2 


Notes: Only those informants identified by two or more grantees are included in this table. 


Timeline 

The timelines utilized by the grantees in their plans are summarized 
in Figure 1. Baseline, 6-month, and 12-month data collection points were 
the most commonly utilized by grantees in their timelines. I n terms of number 
of intervals utilized in the plan, four of the grantees collected data at four 
points in time; one grantee collected data at two and one grantee at eight 
points. While most grantees (four) tied their plans to time since entry into 
the program, one grantee tied follow-up data collection to leaving the program. 
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Figure 1 

Timelines for Measuring Outcomes 


Baseline: 8 grantees 


3 months: 2 grantees 

6 months: 6 grantees 

9 months: 1 grantee 

12 months: 6 grantees 

18 months: 2 grantees 

24 months: 4 grantees 

, 1 i I 1 ^r~ 



Upon leaving the program: 3 grantees 
6-12 months after completing program: 1 grantee 
Annual review of secondary data: 1 grantee 
36 and 48 months from baseline: 1 grantee 


Specific Measures 

The measures most commonly included in these plans are 
summarized in Table 5. Six of the eight grantees developed (or intended to 
develop) local instruments to use in their plans. These instruments were 
largely intended to measure outcomes not normally addressed in commonly 
used instruments (e.g., spirituality), tap into secondary data to measure 
community impacts (e.g., decreased domestic violence), or to measure 
domains from an AI/AN perspective. Only five commonly used individual/ 
family-based measures were selected by more than one grantee: the Behavior 
and Emotional Rating Scale, the Child Behavior Checklist (and related 
measures such as the Youth Self-Report), the Child and Adolescent Functional 
Assessment Scale, Client Satisfaction Questionnaire, and the Family and 
Youth Satisfaction Questionnaire. The Voices of Indian Teens Survey was 
the only measure identified by more than one grantee that had been developed 
specifically for use with Al’s. Community Readiness (Oetting, Jumper- 
Thurman, Plested, & Edwards, 2001) was the most common approach to 
examining community-level outcomes without relying on secondary data. 
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Also notable is that of the eight measures chosen by two or more grantees, 
four were measures included in the mandatory evaluation plan for the CMHS' 
Children's Mental Health Systems grants (SAMHSA, 1998b). 

Table 5 

Key Measures I ncluded in the Circles of Care Outcomes Measurement Plans 


Domains Number of Grantees I ncluding 

this Domain in their Plan 


Locally-developed measures 6 

Child Behavior Checklist and Related Measures 5 

Behavior and Emotional Rating Scale 3 

Child and Adolescent Functional Assessment Scale 3 

Community Readiness Measures 3 

Client Satisfaction Questionnaire - 8 2 

Family and Youth Satisfaction Questionnaires 2 

Voices of Indian Teens Survey 2 


Conclusions and I mplications 

The CoC grantees developed outcomes measurement plans that, 
while affected by the demands of potential funding mechanisms, also 
incorporated instrumentation that would allow them to focus on the domains, 
assessment levels, assessment approaches, and timelines most appropriate 
to their strategic plans. I ndeed, the process and products of the CoC plans 
for measuring outcomes have important implications for communities, clinical 
programs, evaluators, and policymakers. 

For communities and the clinical programs that serve them, the CoC 
grantees demonstrated that Outcome Measurement Plans can focus on both 
individual and family strengths and problems. I ndeed, as a group the CoC 
grantees identified a menu of measures that approached outcomes from 
these different perspectives. Such a balanced approach to measuring 
outcomes should provide a more complete assessment of the progress a 
child and family make while they are receiving services and afterwards, and 
should serve as a model for other systems of care. 

Furthermore, these results underscore the importance community 
members place on measuring community level outcomes. Thus, there is an 
expectation among community members that mental health services will not 
only provide positive outcomes for children, adolescents, and families who 
receive these services, but that the impact of services will extend to the 
greater community as well. Not only should children and adolescents who 
receive mental health services be more likely to graduate from high school, 
but effective mental health services should raise the graduation rate for 
entire schools. Thus, an effective system of care was viewed by the CoC 
communities as a healing process not only for individuals and families, but 
for communities as well. 
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For evaluators, the Outcome Measurement Plans produced by the 
CoC grantees demonstrate the importance of a participatory approach to 
developing these plans. Given the constraints already placed on programs 
and communities by policymakers and funders, evaluators need to be 
particularly attuned to their program and community partners and work to 
identify existing measures that meet the community's visions of positive 
outcomes rather than relying on those problem-focused measures that are 
typically employed in mental health programs. Given the likely possibility 
that no existing measures will be completely consistent with this vision, 
evaluators should be prepared to work with their partners to develop 
measures and measurement approaches that fill in the gaps that will almost 
certainly exist. A full appreciation of the scope, strengths, and weaknesses 
of existing measures, the process involved in developing community-specific 
measures, and the use of implicit measurement techniques ( Brook & Cleary, 
1996) for particularly complex areas of assessment (e.g., whether an child, 
adolescent, and family are "in balance," and how they are "out of balance") 
is a particularly important ability for evaluators to possess. 

There are two important lessons for policymakers in the Outcomes 
Measurement Plans produced by the CoC grantees. First, these plans serve 
as both a wonderful example of what communities are capable of when 
given the time and necessary fiscal and technical support to develop model 
programs. The resultant plans for measuring outcomes are particularly 
impressive in their comprehensiveness, the ways they reflect community 
values and beliefs, and their responsiveness to clinical perspectives of mental 
health service delivery. As such, the process for these developing plans, 
embodied in the CoC approach to strategic planning and program evaluation, 
are a model for similar efforts in both AI/AN and non-AI/AN communities 
alike. 

Second, these plans also demonstrate the power the funders hold 
in shaping the entire discussion on measuring Outcomes. The more funders 
specify the use of specific outcomes measures, the less communities and 
clinical programs will pursue innovative approaches to measurement. Thus, 
funders must be thoughtful in balancing their need for commonality in outcome 
measurement in the programs they support and the need for communities 
and service systems to measure the outcomes that reflect the values and 
beliefs of the communities they serve. 

Douglas K. Novins, M. D. 

Director, Circles of Care Evaluation Technical Assistance Center 
American I ndian and Alaska Native Programs 
Nighthorse Campbell Native Flealth Building 
P.O. Box 6508, Mail Stop F800 
Aurora, CO 80045-0508 
Tele: (303) 724-1414 
FAX: (303) 724-1474 
Email: douglas.novins@uchsc.edu 
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Footnote 


1 Eight of the Nine Circles of Care grantees provided Outcomes Measure- 
ment Plans for this report. 



